Summary

Adds per-upstream circuit breaking to the gateway, preventing cascading failures when a backend becomes unhealthy. Tracks upstream failures on a resilience4j-style three-state machine (CLOSED → OPEN → HALF_OPEN → CLOSED), trips on either consecutive-failure or failure-rate thresholds, and short-circuits checkouts with 503 Service Unavailable while the circuit is OPEN. A separate retry budget caps the fraction of concurrent upstream work that may be retries, bounding the retry-storm amplification factor even when individual retries pass the breaker gate.

What's in this PR

Config schema

• CircuitBreakerConfig struct in include/config/server_config.h (12 fields: enabled, dry_run, thresholds, window, half-open budget, open-duration bounds, retry-budget tuning).
• Nested into UpstreamConfig. The UpstreamConfig::operator== equality operator EXCLUDES circuit_breaker because those fields are live-reloadable — topology fields (name, host, port, tls, pool, proxy) remain restart-only.
• JSON parse with strict per-field type validation (is_number_integer / is_boolean — rejects 1.9 → 1 and true → 1 silent coercions).
• 13 validation rules in ConfigLoader::Validate; upper bounds on consecutive_failure_threshold (≤10k), minimum_volume (≤10M), permitted_half_open_calls (≤1k).
• ConfigLoader::ValidateHotReloadable(config, live_upstream_names) — narrowed gate run on every SIGHUP / HttpServer::Reload. Validates breaker fields ONLY for upstreams that exist in the running server (matching the apply-scope of CircuitBreakerManager::Reload); rejects duplicate names unconditionally on the new file. Hard-rejects on failure rather than downgrading to a warn — the wider Validate downgrade-to-warn is preserved for restart-only fields only.
• Round-trip ToJson serialization.

Core state machine + sliding window

include/circuit_breaker/:

• circuit_breaker_state.h — State, Decision { ADMITTED, ADMITTED_PROBE, REJECTED_OPEN, REJECTED_OPEN_DRYRUN }, FailureKind, StateTransitionCallback.
• circuit_breaker_window.h/.cc — time-bucketed sliding window (ring of per-second buckets, lazy advance, dispatcher-thread-local). Constructor clamps non-positive window_seconds to 1.
• circuit_breaker_slice.h/.cc — per-dispatcher breaker slice with:
  • Dual trip paths (consecutive-failure OR rate-with-min-volume).
  • Lazy OPEN → HALF_OPEN on next TryAcquire.
  • Exponential open duration (base << consecutive_trips, capped at max); ComputeOpenDuration clamps non-positive / inverted bounds at use.
  • Bounded HALF_OPEN probes via half_open_admitted_ (monotone per-cycle counter, not inflight — prevents slot-reuse after one probe completes).
  • Snapshot of permitted_half_open_calls at cycle entry so a mid-cycle reload can't change the budget for the running cycle.
  • Dry-run mode (returns REJECTED_OPEN_DRYRUN; caller proceeds).
  • Generation tokens split by admission domain (closed_gen_ / halfopen_gen_) — stale reports drop silently; window-resize bump doesn't strand in-flight probes.
  • TryAcquire() returns Admission { Decision, uint64_t generation }; Report{Success,Failure,Neutral} takes the admission generation.
  • ReportNeutral — slot-release path for admissions that terminate locally (POOL_EXHAUSTED, shutdown, client disconnect) without counting as success or failure.
  • Config hot-reload preserving live state on threshold-only edits; full reset on enabled toggle; window-resize also resets consecutive_failures_.
  • Synthetic OPEN→OPEN callback with trigger="dry_run_disabled" fired by Reload when dry_run flips true→false on a slice that's still OPEN — lets the host's transition handler flush queued shadow-mode waiters.
  • Disabled fast path: single if (!config_.enabled) return ADMITTED; early return, zero atomic traffic when off.
  • Time source injection for deterministic tests.
  • Public accessors: IsOpenDeadlineSet(), config(), NextOpenDurationMs().

Host / Manager / RetryBudget

• retry_budget.h/.cc — RetryBudget class. RAII InFlightGuard for per-attempt bookkeeping. CAS loop TryConsumeRetry (concurrent retries cannot race past the cap) with a non-retry denominator (cap = max(min_conc, (in_flight - retries_in_flight) * percent / 100)), so in steady state the effective retry fraction matches the configured percent rather than drifting above it. ComputeCap() observability accessor.
• circuit_breaker_host.h/.cc — CircuitBreakerHost owns N slices (one per dispatcher partition) + one shared RetryBudget. Snapshot() aggregates per-slice counters + retry-budget state. Reload() fans out per-slice Slice::Reload calls via Dispatcher::EnQueue. host_label format: service=<svc> host=<h>:<p> partition=<i>.
• circuit_breaker_manager.h/.cc — CircuitBreakerManager keyed by service name. Topology stable post-construction (lock-free GetHost). Constructor validates dispatcher-count vs config partition-count mismatch (throws; skipped when dispatchers is empty for unit-test paths). Reload() serialized by mutex.

Hot-path integration — ProxyTransaction + UpstreamManager + HttpServer

Ownership & wiring:

• HttpServer::circuit_breaker_manager_ — declared AFTER upstream_manager_ so destruction runs breaker-first.
• UpstreamManager::AttachCircuitBreakerManager(raw*) — atomic non-owning pointer (release/acquire).
• HttpServer::MarkServerReady installs a per-slice transition callback capturing (service, dispatcher_index, slice_ptr). Drains the partition wait queue on:
  • CLOSED→OPEN (fresh trip — queued waiters fail fast)
  • HALF_OPEN→OPEN (probe cycle re-tripped — defensively flushes)
  • OPEN→OPEN with trigger="dry_run_disabled" (synthetic signal from Slice::Reload when shadow mode is turned off on a tripped slice)
  • All paths gated on slice_ptr->config().dry_run == false (live read) so shadow mode skips the drain and emits a [dry-run] ... skipping (shadow mode) info log.
• Wired for ALL upstreams regardless of enabled so live reload from enabled=false→true works without re-wiring.

Result codes:

• PoolPartition::CHECKOUT_CIRCUIT_OPEN = -6.
• ProxyTransaction::RESULT_CIRCUIT_OPEN = -7, RESULT_RETRY_BUDGET_EXHAUSTED = -8.

ProxyTransaction hot-path changes:

• slice_ + retry_budget_ resolved once at Start(). Both pointers cached unconditionally when the host exists; per-attempt USE sites live-check slice_->config().enabled so SIGHUP toggling the master switch takes effect on the next retry of an in-flight transaction.
• AttemptCheckout calls ConsultBreaker() at the top; each attempt (first + retries) gets a fresh admission stamped with the slice's current generation.
• Retry-budget gate at AttemptCheckout entry (not in MaybeRetry) so a delayed retry doesn't hold a token during its backoff sleep, matching the "aggregate upstream load" semantic of the cap.
• Retry-budget reject path releases the slice admission (ReleaseBreakerAdmissionNeutral()) before delivering 503 — otherwise a HALF_OPEN probe slot was reserved and would leak forever.
• Cleanup() releases inflight_guard_ alongside the retry token — without this, the failed attempt continued to count against RetryBudget::in_flight_ for the entire backoff sleep, weakening the budget exactly during retry storms.
• ReportBreakerOutcome(result_code) classifies per design §7 and fires BEFORE MaybeRetry at every failure site so the retry's fresh ConsultBreaker sees the latest count.
• ReleaseBreakerAdmissionNeutral() in Cancel() — client-disconnect always neutral (replacement probe slot acceptable; tripping a healthy backend on user-side abandonment would be a DOS vector).

Response factories:

• MakeCircuitOpenResponse() — state-aware Retry-After: OPEN reads stored slice->OpenUntil(); HALF_OPEN uses slice->NextOpenDurationMs() (exponential-backoff aware). Ceil division. Absolute cap 3600s. X-Circuit-Breaker: open|half_open label (distinguishes the two reject paths).
• MakeRetryBudgetResponse() — 503 + X-Retry-Budget-Exhausted: 1 + Connection: close. No Retry-After (budget has no recovery clock).
• Static MakeErrorResponse(RESULT_CIRCUIT_OPEN) fallback emits at least X-Circuit-Breaker: open + Connection: close so even a misuse-from-static-context path stays self-identifying.

Wait-queue drain on trip

• PoolPartition::DrainWaitQueueOnTrip() — dispatcher-thread. Iterates wait_queue_, pops each entry, fires error_callback(CHECKOUT_CIRCUIT_OPEN) on non-cancelled waiters. Skips if shutting_down_. Hoists alive_ against teardown re-entry.
• UpstreamManager::GetPoolPartition(service, index) accessor.

Observability

All events surface through structured logs + a snapshot API. Full log catalog is in docs/circuit_breaker.md §Observability. Highlights:

• CLOSED → OPEN trip at warn: trigger, consecutive_failures, window_total, window_fail_rate, open_for_ms, consecutive_trips.
• OPEN → HALF_OPEN / HALF_OPEN → CLOSED / HALF_OPEN → OPEN at appropriate levels.
• Reject logs — first of cycle at info for the breadcrumb, subsequent at debug. Dry-run rejects at info with [dry-run] prefix.
• retry budget exhausted at warn: service, in_flight, retries_in_flight, cap (via RetryBudget::ComputeCap() accessor).
• circuit breaker config applied at info on every reload.
• circuit breaker dry_run disabled while OPEN ... — flushing wait queue at info when shadow mode is disabled on a tripped slice.
• [dry-run] circuit breaker would drain wait queue on trip — skipping (shadow mode) at info when a trip happens with shadow mode active.
• PoolPartition draining wait queue on breaker trip at info with queue_size.

CircuitBreakerManager::SnapshotAll() returns per-host rows with per-slice counters + host aggregates + retry-budget state. A future /admin/breakers HTTP endpoint that JSON-serializes this is planned but not yet exposed.

Hot-reload

• HttpServer::Reload invokes circuit_breaker_manager_->Reload(new_config.upstreams) unconditionally — idempotent when no CB fields changed.
• HttpServer::Reload also runs ConfigLoader::ValidateHotReloadable(new_config, live_upstream_names) BEFORE applying — symmetric with the CLI SIGHUP path. Library-API callers can't bypass the gate.
• Per-slice Slice::Reload is enqueued on the owning dispatcher so config mutations happen on the correct thread.
• Live state preserved on threshold-only edits; silent full reset on enabled toggle.
• Topology warn rephrased: "upstream topology changes require a restart to take effect (circuit-breaker field edits, if any, were applied live)".
• upstream_configs_ baseline pinned to live state — only updated when topology-equal, so subsequent timer-cadence recomputation tracks live pool timeouts, not staged-but-inactive values.
• Dry-run flip while OPEN: synthetic OPEN→OPEN transition callback fires with trigger "dry_run_disabled" so the partition's wait queue gets flushed once enforcement is back on.

Development review history

The feature was built iteratively across 8 implementation phases plus extensive review-round revisions. Major review-caught regressions are captured as pitfall entries for rules and locked in by regression tests in the suites listed below. Highlights:

Slice / state-machine rounds: pre-increment shift, Report* state guard, HALF_OPEN saw_failure short-circuit, Reload-across-enabled-toggle reset, saw_failure counter misclassification, generation token for stale-report drop, OpenUntil() cleared in HALF_OPEN, window-resize generation bump, domain-split generation (closed_gen_ / halfopen_gen_), orphaned consecutive_failures_ reset, probe budget snapshot at cycle entry, << 0 crash clamp, JSON strict type accessors, ComputeOpenDuration clamps, half_open_admitted_ monotone counter, ReportNeutral, main.cc reload config save/restore.

Hot-path integration rounds (most recent):

Tests (+108 total, 365 → 522)

Config (test/config_test.h)

Defaults, JSON parse, partial block, round-trip, equality (CB excluded from UpstreamConfig::operator==), 13 validation cases, 3 type-strictness cases.

Circuit-breaker test suites (test/circuit_breaker_*_test.h)

Build system

• Makefile: CIRCUIT_BREAKER_SRCS = 5 .cc files; CIRCUIT_BREAKER_HEADERS = 6 .h files; TEST_HEADERS includes all 6 circuit_breaker*_test.h files plus the core unit suite.
• ./test_runner circuit_breaker (or -B) runs every circuit-breaker suite.

Documentation

• Public user guide: docs/circuit_breaker.md — configuration fields, client-facing responses, hot-reload semantics, observability.

Test plan

• make clean && make -j4 produces a clean build.
• ./test_runner passes all 522 tests.
• ./test_runner circuit_breaker (or -B) runs the circuit-breaker suites in isolation.
• With circuit_breaker.enabled=false (the default), there is no behavioral change to production traffic — hot path is a single branch read against a nullptr slice or the disabled fast path.
• SIGHUP with a pure CB-field edit applies live and emits "circuit breaker config applied" without a restart warn. Topology edits emit "upstream topology changes require a restart to take effect (circuit-breaker field edits, if any, were applied live)". Invalid CB tuning is hard-rejected at the reload gate.
• SIGHUP with dry_run=true → false on an OPEN slice flushes the partition wait queue so re-enabled enforcement actually takes effect.